Mixed reality system for context-aware virtual object rendering

ABSTRACT

A computer-implemented method in conjunction with mixed reality gear (e.g., a headset) includes imaging a real scene encompassing a user wearing a mixed reality output apparatus. The method includes determining data describing a real context of the real scene, based on the imaging; for example, identifying or classifying objects, lighting, sound or persons in the scene. The method includes selecting a set of content including content enabling rendering of at least one virtual object from a content library, based on the data describing a real context, using various selection algorithms. The method includes rendering the virtual object in the mixed reality session by the mixed reality output apparatus, optionally based on the data describing a real context (“context parameters”). An apparatus is configured to perform the method using hardware, firmware, and/or software.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Pat. Application No.16/212,438, filed Dec. 6, 2018, which is a continuation of InternationalApplication PCT/US2017/035965 filed Jun. 5, 2017, which claims priorityto U.S. provisional patent application Serial No. 62/346,413 filed Jun.6, 2016, which applications are incorporated herein by reference intheir entireties.

FIELD

The present disclosure relates to methods and apparatus forconfiguration, by a computer, of digital data for virtual reality oraugmented reality output that is geometrically and chronologicallycoordinated with user context-relevant parameters determined from sensordata responsive to the user’s physical environment.

BACKGROUND

“Virtual reality” is a term that has been used for various types ofcontent that simulates immersion in a three-dimensional (3D) world,including, for example, various video game content, and animated filmcontent. In some types of virtual reality, a user can navigate through asimulation of a 3D environment generated based on the computer model, bycontrolling the position and orientation of a virtual camera thatdefines a viewpoint for a 2D scene that is displayed on atwo-dimensional display screen. A variation of these technologies issometimes called “augmented reality.” In an augmented reality setup, thedisplay technology shows a combination of the user’s surroundings thatis “augmented” by one or more digital objects or overlays. Augmentedreality content may be as simple as textual “heads up” information aboutobjects or people visible around the user, or as complex as transformingthe entire appearance of the user’s surroundings into a fantasyenvironment that corresponds to the user’s real surroundings. Virtualreality (VR) and augmented reality (AR) when applied to mix real objectsfrom the user’s surroundings with virtual objects are collectivelyreferred to herein as “mixed reality.”

Virtual reality (VR) and augmented reality (AR) have been applied tovarious types of immersive video stereoscopic presentation techniquesincluding, for example, stereoscopic virtual reality headsets. Headsetsand other presentation methods immerse the user in a 3D scene. Lenses inthe headset enable the user to focus on a lightweight split displayscreen mounted in the headset only inches from the user’s eyes.Different sides of the split display show right and left stereoscopicviews of video content, while the user’s peripheral view is blocked. Inanother type of headset, two separate displays are used to showdifferent images to the user’s left eye and right eye respectively. Inanother type of headset, the field of view of the display encompassesthe full field of view of eye including the peripheral view. In anothertype of headset, an image is projected on the user’s retina usingcontrollable small lasers, mirrors or lenses. Either way, the headsetenables the user to experience the displayed virtual reality contentmore as if the viewer were immersed in a real scene. In the case ofaugmented reality (AR) content, the viewer may experience the augmentedcontent as if it were a part of, or placed in, an augmented real scene.A similar effect can be achieved using virtual reality headsets byincluding real-time image data from a stereoscopic camera pair mountedto the user’s headset in the VR feed, and rendering virtual objectsmixed with the real-time image feed.

These immersive effects may be provided or enhanced by motion sensors inthe headset that detect motion of the user’s head, and adjust the videodisplay(s) accordingly. By turning his head to the side, the user cansee the virtual reality scene off to the side; by turning his head up ordown, the user can look up or down in the virtual reality scene. Theheadset may also include tracking sensors that detect position of theuser’s head and/or body, and adjust the video display(s) accordingly. Byleaning or turning, the user can see the virtual reality scene from adifferent point of view. This responsiveness to head movement, headposition and body position greatly enhances the immersive effectachievable by the headset. The user may be provided the impression ofbeing placed inside or “immersed” in the virtual reality scene. As usedherein, “immersive” generally encompasses both VR and AR.

Immersive headsets and other wearable immersive output devices areespecially useful for game play of various types, which involve userexploration of a modelled environment generated by a rendering engine asthe user controls one or more virtual camera(s) using head movement, theposition or orientation of the user’s body, head, eye, hands, fingers,feet, or other body parts, and/or other inputs. To provide an immersiveexperience, the user needs to perceive a freedom of movement that is insome way analogous to human visual perception when interacting withreality. Content produced for VR can provide this experience usingtechniques for real-time rendering that have been developed for varioustypes of video games. The content is may be designed as athree-dimensional computer model with defined boundaries and rules forrendering as video output. This content can be enhanced by stereoscopictechniques to provide stereoscopic output, sometime referred to as “3D,”and associated with a VR application that manages the rendering processin response to movement of the VR headset, to produce a resulting VRexperience. The user experience is very much like being placed inside arendered video game.

However, use of VR and AR for immersive entertainment is limited tospecific contexts, for example while the user is stationary (e.g.,seated in a chair) or confined to moving in a small space defined by asensor array. In such limited contexts, there is little opportunity fora mixed reality application to incorporate information in the userexperience as the user’s context changes, because there is little changethat occurs in that context during the period of play.

It would be desirable, therefore, to develop new methods and other newtechnologies for configuring digital content for VR and AR use, thatovercome these and other limitations of the prior art and enhance theappeal and enjoyment of mixed reality content for new immersivetechnologies such as VR and AR.

SUMMARY

This summary and the following detailed description should beinterpreted as complementary parts of an integrated disclosure, whichparts may include redundant subject matter and/or supplemental subjectmatter. An omission in either section does not indicate priority orrelative importance of any element described in the integratedapplication. Differences between the sections may include supplementaldisclosures of alternative embodiments, additional details, oralternative descriptions of identical embodiments using differentterminology, as should be apparent from the respective disclosures.

In an aspect of the disclosure, a computer-implemented method includesimaging a real scene encompassing a user wearing a mixed reality outputapparatus. The method may include determining data describing a realcontext of the real scene, based on the imaging. The method may includeselecting a set of content from a content library, based on the datadescribing a real context. The set of content may be, or may include,data that enables rendering of a virtual object. The method may includerendering a virtual object described by the set of content in the mixedreality session by the mixed reality output apparatus, based on the datadescribing a real context. For example, lighting parameters may beautomatically taken or derived from the context parameters by thecomputer processor.

The foregoing method may be implemented in any suitable programmablecomputing apparatus, by provided program instructions in anon-transitory computer-readable medium that, when executed by acomputer processor, cause the apparatus to perform the describedoperations. An apparatus may include a computer or set of connectedcomputers that is used in mixed reality production or is installed in auser head set. Other elements of the apparatus may include, for example,an audio output device, and a user input device, which participate inthe execution of the method. An apparatus may include a virtual realitydevice, such as a headset or other display that reacts to movements of auser’s head or body to provide the impression of being placed inside ofthe rendered scene in which a game is played or narrative content isbeing displayed.

To the accomplishment of the foregoing and related ends, one or moreexamples comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative aspectsand are indicative of but a few of the various ways in which theprinciples of the examples may be employed. Other advantages and novelfeatures will become apparent from the following detailed descriptionwhen considered in conjunction with the drawings and the disclosedexamples, which encompass all such aspects and their equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify like elements correspondingly throughout thespecification and drawings.

FIG. 1 is a schematic block diagram illustrating aspects of a system andapparatus for the production and configuration of digital data for mixedreality output coupled to a distribution system.

FIG. 2 is a schematic block diagram illustrating more detailed aspectsof an apparatus for outputting mixed reality content.

FIG. 3 is a schematic diagram illustrating aspects of viewing mixedreality content from the perspective of different viewers.

FIG. 4 is a concept diagram illustrating elements of a system forcontext determination and content selection for a mixed-reality session.

FIG. 5 is a block diagram illustrating aspects of a system performingreal context description determination using input from a sensor array.

FIG. 6 is a block diagram illustrating aspects of system performingcontent selection for a mixed reality process based on a data describinga real context.

FIG. 7 is a schematic diagram illustrating components of a stereoscopicdisplay device for providing an immersive mixed reality experience.

FIG. 8 is a concept diagram illustrating elements of a systemintegrating virtual content with real environment input to provide amixed reality experience.

FIG. 9 is a flow chart illustrating a method for mixed reality contentselection.

FIGS. 10-12 are flow charts illustrating further optional aspects oroperations of the method diagrammed in FIG. 9 .

FIG. 13 is a conceptual block diagram illustrating components of anapparatus or system for mixed reality content selection.

DETAILED DESCRIPTION

Various aspects are now described with reference to the drawings. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide a thorough understanding ofone or more aspects. It may be evident, however, that the variousaspects may be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform to facilitate describing these aspects.

An illustrative system 100 for production and distribution of mixedreality content is shown in FIG. 1 . The system 100 may include a set102 of production activities that produce assets that are shared andused in different ways across related different versions (e.g.,immersive and non-immersive versions) of underlying creative content.Creative content includes, for example, video data collected by variouscamera systems 112, 112, audio data collected and/or generated by audiosubsystems (not shown), and computer modeling/animation data created andarranged from various modeling/animation subsystems 108, 110. Creativecontent may be stored in a data store 106. It should be appreciated thatthe system may include several different data stores (not shown). Aproduction server component 104, which may comprise a family ofproduction applications operating over a computer network, may accessdata in the data store 106 under control of various production staffcontrolling the production process via multiple access terminals 118,116. The number of components shown in system 100 is merelyillustrative. It should be appreciated that a typical feature film orother studio production system will typically include a much largernumber of components than illustrated. Creative and technical directorsoversee the assembly of creative content from the various data sources,configured for immersive output devices and more traditionalnon-immersive devices.

Digital content produced by the system may include various versions ofthe same or related creative titles, for example, immersive mixedreality AR and VR versions in the nature of a video game or interactivenarrative; various non-immersive versions such as, for example, a 2Dtheater version, a 2D home theater version, a mobile device version, astereoscopic 3D version for one or more of theater, home or mobiledevices; combination immersive and non-immersive versions such as, forexample, a VR version for an in-theater experience, in conjunction withsupporting 2D or stereoscopic 3D content, a VR version for home use,likewise for use with non-immersive content; an AR version forsupplementing non-immersive content in a theater, an AR version forsupplementing non-immersive content in a home theater environment or ina mobile device format. Finished productions in each of the variousversions may be provided to a home distribution server 120 which maystore the different versions in a content data store (not shown) inassociation with metadata for managing use and distribution. A least oneset of consumers may receive multiple versions of immersive andnon-immersive content in a single digital content (media) package,whether stored under control of a network served 120, or locally on acomputer-readable medium such as an optical disc or memory device.

Different distribution channels each assigned its own server resourcesmay be used to provide content to different sets of end users. Forexample, a cinema distribution server 130 may distribute immersive andconventional content to cinemas for public performance. For illustrativeclarity, one cinema 140 of potentially many cinemas is diagrammed. Eachcinema 140 may include at least one server 134 used to distributedigital content to one or more theaters each hosting a performance. Eachtheater (or the theater, if only a single theater is served by theserver 143) includes a cinema screen 136 and one or more viewers eachwearing an immersive content consumption device, 132, 138, for example,a VR visor or AR headset. The same underlying audio-video program maythereby be distributed in different versions for home and cinema use.Both home and cinema versions may include technical elements thatcoordinate different immersive devices contemporaneously playing theaudio-video program in an immersive format. In addition, both versionsmay include elements that coordinate play of immersive content withcontemporaneous or non-contemporaneous content playing on a 2D screen.

Interactive mixed reality content may be designed for rapidly changingcontexts, for example, a user strolling through a park or urbanenvironment, and is not generally designed to be coordinated withnon-immersive content. However, interactive mixed reality content maysometimes be coordinated with non-immersive content for clips of limitedduration appearing in the user’s environment. For example, a mixedreality AR application may detect that the user is viewing a videobillboard or other image display appearing in the user’s realenvironment, and configure the mixed reality content based on thecontent appearing on the real display in some way. For example, themixed reality application may be configured to replace the contentappearing on a real video display or static image display with asubstitute video or static image in the mixed reality session. In analternative, or in addition, the mixed reality application may beconfigured to cause a virtual object or character that relates tocontent appearing on the real display to appear in the mixed realitysession. For example, if the real content includes advertising for aparticular product, the application may insert an animated mascotcharacter for the product (or for a competing product) in the mixedreality session. For further example, if the real content depicts aparticular scene, object, theme or character, the application may insertvirtual content that enhances, parodies, or contradicts the realcontent.

In some embodiments, a media package holding coordinated immersive andnon-immersive content may be, or may include, a single computer-readablemedium (for example, an optical disc medium or FLASH memory device) inwhich packaged digital content is stored together. Distribution of anon-transitory, tangible and portable storage medium may reduce networkbandwidth demands and ensure reliable and seamless access to densedigital content by the consumption device. In some embodiments, rapiddistribution to tangible media may be accomplished by distribution fromselected kiosks holding electronic copies of digital content for writingto digital copies. In an alternative, such kiosks may take advantage ofhigh-bandwidth connections to obtain the electronic content fordistribution. In other embodiments, including for example for cinemadistribution, the electronic content may be transmitted over acommunications network and/or computer network and stored directly on amemory device or medium connected to or integrated with a client devicethat will participate in playback of the received content. Mixed realityinteractive versions may be distributed in conjunction with distributionof other versions. For example, a user attending a cinematicpresentation of a feature film may be provided with an option todownload a related mixed reality interactive game from a cinemadistribution server to a head set data storage unit while watching thefeature film, or to receive the mixed reality game on a portable memorydevice, or to receive a stream that accompanies the cinematicpresentation, adding mixed reality assets to enhance the otherwisecinematic-only presentation.

Referring to FIG. 2 , aspects of a content consumption device 200 forconsuming mixed reality immersive content are illustrated. Severalviewers of a home theater or cinema presentation may be equipped withthe content consumption device. The apparatus 200 may include, forexample, a processor 202, for example a central processing unit based on80x86 architecture as designed by Intel™ or AMD™, a system-on-a-chip asdesigned by ARM™, or any other suitable microprocessor. The processor202 may be communicatively coupled to auxiliary devices or modules ofthe 3D environment apparatus 200, using a bus or other coupling.Optionally, the processor 202 and some or all of its coupled auxiliarydevices or modules (examples of which are depicted at 204-216) may behoused within or coupled to a housing 218, for example, a housing havinga form factor of a wearable googles, glasses, or visor, or other formfactor.

A user interface device 204 may be coupled to the processor 202 forproviding user control input to an immersive mixed reality contentdisplay process operated by a mixed reality immersive display engineexecuting on the processor 202. User control input may include, forexample, selections from a graphical user interface or other input(e.g., textual or directional commands) generated via a touch screen,keyboard, pointing device (e.g., game controller), microphone, motionsensor, camera, or some combination of these or other input devices.Control input may also be provided via a sensor 206 coupled to theprocessor 202. A sensor may comprise, for example, a motion sensor(e.g., an accelerometer), a position sensor, a camera or camera array(e.g., stereoscopic array), a biometric temperature or pulse sensor, awind speed and direction sensor, a touch (pressure) sensor, analtimeter, a location sensor (for example, a Global Positioning System(GPS) receiver and controller), a distance sensor (e.g., electronic tapemeasure), a proximity sensor, a motion sensor, a smoke or vapordetector, a gyroscopic positon sensor, a radio receiver, a multi-cameratracking sensor/controller such as, for example, available fromMicrosoft™ under the brand Kinect™, an eye-tracking sensor, a microphoneor a microphone array. The sensor 206 may detect a user context, meaningan identify, position, size, orientation and movement of the user’sphysical environment and of objects in the environment, motion or otherstate of a user interface display, for example, motion of avirtual-reality headset, or the bodily state of the user, for example,facial expression, skin temperature, pupil dilation, respiration rate,muscle tension, nervous system activity, or pulse.

The device 200 may optionally include an input/output port 208 coupledto the processor 202, to enable communication between a mixed realityengine and a computer network, for example a cinema content server orhome theater server. Such communication may be used, for example, toenable multiplayer VR or AR experiences, including but not limited toshared immersive experiencing of cinematic content. The system may alsobe used for non-cinematic multi-user applications, for example socialnetworking, group entertainment experiences, instructional environments,video gaming, and so forth.

A display 220 may be coupled to the processor 202, for example via agraphics processing unit (not shown) integrated in the processor 202 orin a separate chip. The display 210 may include, for example, a flatscreen color liquid crystal (LCD) display illuminated by light-emittingdiodes (LEDs) or other lamps, a projector driven by an LCD display or bya digital light processing (DLP) unit, a laser projector, or otherdigital display device. The display device 210 may be incorporated intoa virtual reality headset or other immersive display system. Videooutput driven by a mixed reality immersive display engine operating onthe processor 202, or other application for coordinating user inputswith an immersive content display and/or generating the display, may beprovided to the display device 210 and output as a video display to theuser (also referred to herein as the “player”). Similarly, anamplifier/speaker or other audio output transducer 222 may be coupled tothe processor 202 via an audio processing system. Audio outputcorrelated to the video output and generated by the mixed realitydisplay engine or other application may be provided to the audiotransducer 222 and output as audible sound to the user.

The 3D environment apparatus 200 may further include a random accessmemory (RAM) 214 holding program instructions and data for rapidexecution or processing by the processor during controlling a modeled 3Dobjects or environment. When the device 200 is powered off or in aninactive state, program instructions and data may be stored in along-term memory, for example, a non-volatile magnetic, optical, orelectronic memory storage device 216. Either or both of the RAM 214 orthe storage device 216 may comprise a non-transitory computer-readablemedium holding program instructions, that when executed by the processor202, cause the device 200 to perform a method or operations as describedherein. Program instructions may be written in any suitable high-levellanguage, for example, C, C++, C#, or Java™, and compiled to producemachine-language code for execution by the processor. Programinstructions may be grouped into functional modules, to facilitatecoding efficiency and comprehensibility. It should be appreciated thatsuch modules, even if discernable as divisions or grouping in sourcecode, are not necessarily distinguishable as separate code blocks inmachine-level coding. Code bundles directed toward a specific type offunction may be considered to comprise a module, regardless of whetheror not machine code on the bundle can be executed independently of othermachine code. In other words, the modules may be high-level modulesonly.

FIG. 3 illustrates aspects of mixed reality content using AR or VR in aviewing space 300 shared by multiple persons 314, 316. A first person314 wearing an AR or VR headset views a real content object 310 (“tree”)in the viewing space. If using VR equipment, the real object 310 may becaptured by a camera of camera array mounted to the headset anddisplayed on a VR display screen. If using an AR visor or glasses, theperson 314 sees the object 310 through a transparent portion of the VRequipment. In addition, the first person sees a virtual dragon 320standing in front of the tree 310. A second person 316 is viewing theobject 310 with “naked eyes” and no equipment. The second person 316sees nothing except the actual physical surroundings (e.g., object 310)in the area surrounding the screen 302.

FIG. 4 shows a system 400 of cooperating components for providing acontext-responsive mixed reality output for at least one user 401 whointeracts with a mixed reality session 410 operated by at least onecomputer processor via a user interface 403. The system 400 may includealgorithms and processes 404, 406 executed by, or in connection with, amixed-reality output device 402 operated by a user. The processes 404,406 process and apply sensor data from a sensor array 408 that collectsdata from the user’s physical environment. The process 404 extractscontext information relevant to a mixed reality session 410 executing onor in connection with the output device 402, or that will be initiatedon the device 402. The selection process 406 receives context relevantparameters from the context determination process 404, and uses theparameters to select mixed-reality objects from a library residing in acomputer memory, for example, in a networked cloud data structure 434.The processes 404, 406 and 410 may execute in a processor or processorof a local mixed reality client worn by the user 401, or by somecombination of local client processors and processors that are locallyor remotely networked to the local client via any suitable computernetwork or combination of computer networks.

For purposes of the present disclosure, the executable system ofalgorithms and processes 404, 406 are collectively called a “MixedReality Context Engine” or “MRCE.” The MRCE uses the sensor data fromthe array 408 to obtain context parameters (also called context-relevantparameters) that can be used in the mixed-reality session 410 to selectand conform objects and events occurring in the mixed reality sessionwith objects and events in the user’s physical environment 412. Thecontext parameters 428 may be selected and determined by the MRCE 404,406 based on the sensor data obtained from the sensor array 408 that issensing the current physical environment 412, and on current sessioncontext information 432 from the mixed reality session 412. Sessioncontext information 432 may be determined directly from a known memorystate of the mixed reality session 410, based on data provided to theoutput device 402. In the alternative, or in addition, if the selectionprocess 406 does not have direct access to the session state 410, objectrecognition software and/or image analysis software can also providecontext information based on output (e.g., audio and video output) froma mixed reality session 410. In whatever manner the session contextinformation 432 is obtained, it may be used in conjunction with sensoroutput to select content for the mixed reality session 410, as furtherdescribed below.

The context parameters 428 may be used, for example, to select a newmixed-reality object or effect from a library 414 of mixed-realityobjects or effects. The mixed reality object or effect may bepre-recorded as a rendered object, for static objects. More generally,the mixed reality object or effect may be preconfigured as a renderableobject, and rendered using inputs that are selected based on the currentcontext. Rendering inputs may include, for example, a characteranimation sequence or set of rules for animating the object in mixedreality, lighting and viewpoint definitions, rendering engineconfiguration, and many other settings as known in the digital renderingarts. In addition, or in an alternative, model attributes besidesanimation, object scale and motion may be algorithmically created on thefly. For example, the object’s appearance and skin may be generated oraltered based on an algorithm that takes as inputs context informationand character components, and outputs a character or other object thatis customized for the current context in both appearance and behavior. Asession module may compile the custom object and insert it into themixed reality session 410, or use the object to initiate a newmixed-reality session. For example, if the sensors indicate that theuser is in a location for which historical mixed-reality data exists,the session may select one or more objects or characters that arerelevant to the location generically or specifically. Inserting amixed-reality animation of an un-identifiable baseball player when thesensors indicate that the user is located in or near a baseball diamondis an example of generic relevance. Inserting an animated historical orpublic figure when the user 401 is located in a place of significance tothe figure is an example of specific relevance.

The selection process 406 may use different selection algorithms forselecting content from the library 414 based on what is happening in themixed reality session 410 as determined from session context information432. For example, if the user’s avatar is located in virtualdragon-infested territory, the process may use dragon selectionalgorithms designed to select dragon models from the models category416, dragon appearance characteristics from the skins category 418, andother dragon-related session objects from other categories. In addition,or in the alternative, the selection process 406 may use differentselection algorithms for selecting content from the library 414 based onphysical environment parameters from the physical context determinationprocess 404. For example, if the process 404 indicates that the user iswalking through wooded parkland, the selection process 406 may selectcontent from the library based on a “woodland” set of algorithms, whichselect library content appropriate for wooded areas. In alternativeembodiments, a selection algorithm may select content based on amulti-factor priority ranking that includes both physical context andvirtual context factors, so that both play a role in content selectionthat may vary based on contextually-determined weighting factors.

For further example, the context parameters 428 may be used to position,orient, and/or scale a selected mixed-reality object or effect in amixed reality session, based on objects or events sensed in the user’sphysical environment. For example, the parameters may be used to cause amixed-reality object to appear to react to physical objects in thephysical environment, such as, for example, to bounce off of or to stickto physical objects.

For further example, the context parameters 428 may be used to makegeometric adjustments to the shape of a selected mixed-reality object oreffect in a mixed reality session, based on objects or events sensed inthe user’s physical environment, so that the mixed reality object oreffects conform to a desired mixed reality grammar desired for thesession.

In an aspect of the system 400 and related methods, the selection fromthe library may use an artificial intelligence process to select anobject or effect that is relevant to the user’s physical environment butthat is not easily predictable or deterministic, for the same user orbetween different users. For example, different users who are nearby toeach other and operating the same mixed-reality program may seedifferent mixed reality objects or effects. For further example, basedon heuristic feedback 430, the selection process 406 may cause a userreturning to the same place to never see precisely the same object oreffect twice, unless requested. In addition, or in an alternative, thecontextual relation(s) between the environment used to make a selectionmay be varied by a non-deterministic (e.g., random) algorithm orfunction, so that there is always an element of surprise concerning howan object is selected, and consequently, which object or which type ofobject will be selected. At the same time, the selection algorithmshould choose an object with some user-discernable relevance to thephysical place or circumstance. A non-deterministic algorithm mayinclude, for example, random and heuristic elements.

Categories of library content may include, for example, 3D modeling data416, which provides surface-modeling, animation armature, and physicsparameters for virtual objects to be rendered. Another category may be“skins” 418 that define the surface characteristics of the model to berendered, for example, diffuse and specular colors, roughness, and othersurface characteristics. An “action” category 420 may include animationloops for objects and characters. An “audio” category 422 may includeaudio clips or samples, code for generating audio procedurally, filesfor controlling audio parameters such as, for example, volume and stereoseparation, and other audio-related data. A “render settings” category424 may include rendering control parameters, links to executable code,and data for controlling rendering of objects as appropriate for aparticular context. A “grammar” category 426 may include high-levelcodes describing various different types of relationships betweenrendered virtual objects and reference objects in a real scene, of whichexamples are provided elsewhere herein. The selection process 406 maypick one or more objects from each category (or from fewer or additionalcategories) to determine exactly how one or more virtual objects are tobe inserted into a particular mixed reality session.

Referring to FIG. 5 , a real context determination module or process 500may process sensor data from a sensor array 502 to arrive at aparticular context description 528 for any given real environment. Whilevarious different sensors may also be suitable, the depictedseven-sensor array 502 should suffice for supporting determination of acontext description based on object and scene types, scene and objectgeometry, scene conditions, and scene and object identities.

A sonar sensor 504 can be used to determine current distances to objectsin the scene and the relative solidity of the objects. Time lag fromsignal to receipt of echo and the echo profile can be used to constructa scene depth map from the point of view of any acousticemitter/receiver pair. An array of such pair may be used to construct adetailed scene and object model using a photogrammetric process 518.Input from an orientation sensor 506 mounted to the user’s head may beused to set current viewpoint parameters. Orientation sensors mayinclude, for example, a solid-state accelerometer array, mechanical tiltsensor array, or gyroscopic unit. Sonar data can be coupled with visibleand/or infrared images from an array of two or cameras 508 in thephotogrammetric process 518 to construct and maintain a model or scenemap as a user moves through a physical environment.

A scene map may be expressed in any suitable modeling data format, forexample, Virtual Reality Modeling Language (VRML) format, or in aproprietary simplified format. For context determination, it is mainlydesired to know the shape of a scene and objects within it, and therelative solidity of objects in the scene, so it can be determined wherevirtual objects can be positioned and moved without violating a chosengrammar. Detailed model information for rendering may be useful, but isnot required in many context determination applications. Accordingly,for example, a suitable simplified format may express shape informationfor any given object using a set of volumetric geometric shapes eachassociated with a solidity factor, a rigidity factor or profile, a scalefactor and an orientation, with some geometric interrelationship betweeneach shape in the set. These geometric parameter sets can be roughapproximations; for example, a set specifying “5 foot diameter sphere,30% solidity centered atop a cylinder, 4 feet long 0.5 feet diameter100% solidity” may suffice to identify the object as a small tree.

A type recognition process 524 may be used to characterize anyparticular set of shapes as being of a specified type, for example,“tree,” “floor,” “wall,” “ceiling,” “cloud,” “table,” and so forth.Scene types may include various types of outdoor scenes and indoorscenes, most of which can be characterized using a small parameter set.A library of a few hundred basic object types should suffice tocharacterize most common types, keeping in mind that for entertainmentpurposes highly accurate typing is seldom required, while accurategeometric and solidity information may be more important than type forproviding a consistent and credible mixed reality experience. Forexample, it may be more important that a rendered virtual objectconsistently appear to be unable to pass through a solid object in thescene, than for the object to recognize the type of solid object it isnavigating around. Type determination may be purely algorithmic, basedon a real-time comparison between a detected object’s geometricparameter set and geometric parameter envelopes defined by a typelibrary. Fuzzy logic may be used to choose the object types based oninput as described.

Data from the camera array 508 and an array or one or more microphonesmay be used to recognize 520 conditions of the scene, primarily lightingconditions and acoustic conditions. Other conditions of interest mayinclude, for example, atmospheric conditions (e.g., rain, smoke, snow orfog). For weather-related conditions, GPS data from a GPS receiver 514may be compared with published weather reports.

Recognition 522 of particular identities of scenes and objects withinthem can be based on location (e.g., GPS) coordinates from a GPSreceiver 514 (or other triangulating location sensor) and publishedlocation information, or upon object recognition and/or image analysissoftware. For smaller objects, radio frequency tagging may be used, withtags detected by a wireless receiver 512. For example, a mixed realityamusement park might place RFID tags on numerous real items placedthroughout the park, so that each user’s AR or VR equipment can quicklylocate and identify nearby objects based on a unique identifiertransmitted by each tag placed on each respective one of the objects.

Input from type recognition 524, condition recognition 520 and identityrecognition 522 may be integrated and processed by a final real contextdetermination module 526 that produces as output computer-readable datadescribing a real current context, sometime referred to herein as acontext description, or a real context description. The real contextdescription may be placed in a memory location having an address orpointer that is possessed by other modules of the mixed reality process,especially a content selection module. As the context changes throughtime, the context determination module 526 updates the current contextdescription. A context description may include, for example, avolumetric solidity map or model of the physical environment;environmental condition parameters; scene and object types; and sceneand object identities. The priority of the description may be in theorder listed, with the volumetric solidity map having the highestpriority. Condition information is also high priority; condition andsolidity are both needed to implement insertion of mixed reality objectsin a real environment according to a consistent grammar. Type andidentity information is useful for providing a more varied and enrichingexperience, but not as fundamental to providing a basic user experience.

A real context description may be blended with a mixed reality contextdescription and optionally with heuristic feedback from a particularcombination of user and mixed reality session to perform a virtualcontent selection process. By way of example, FIG. 6 shows elements ofone such selection process 600 based on the real context description 528and other inputs. A content selection 603 may include different internalmodules or components. For example, a first filter process 610 parsesthe real context description 528 and uses a set of logical rules (whichmay include fuzzy logic) to eliminate library content from considerationthat is incompatible with the physical parameters of the scene. This maybe conceptually understood as a first filter pass of library content.“Incompatible” may mean that a set of content cannot possibly complywith a specified grammar for the mixed reality session, or may have anyother desired meaning so long as it results in a filtering of availablecontent sets. It should be appreciated that ranking of content sets isequivalent to filtering.

A second filter process 612 may re-rank or further filter the output ofthe first process 610, according to heuristic feedback. Heuristicfeedback may include any user input that bears some logical relationshipto the history (past events) of the current mixed reality session, or ofpast mixed reality sessions. For example, heuristic feedback may be usedto prevent objects from being repeatedly selected at different times toprovide a more varied user experience, or to select objects inconformance with user preference data. In an alternative, or inaddition, the selection process 406 may randomly select a virtual objectto include in the mixed reality session. For example, the selectionprocess may select a class of objects (e.g., “dragons”) based on realcontext, session context, or heuristic feedback, and randomly select aparticular instance of the class (e.g., a particular dragon object).

A third filter process 614 may re-rank or further filter the output ofthe second process 612, based on a current session context. For example,if the user has just finished dispatching a difficult computer-generatedadversary, the process 614 may prioritize celebratory characters orobjects over adversarial ones. The third process 614 selects an objectset based on a final ranking, which is inserted into the mixed realityprocess. The behavior of the inserted object and its eventual exit fromthe mixed reality process may be predetermined, or may be managed in acontext-responsive manner by a “life” module 616, using session contextand any changes in the real context description 528 to determinebehavior and other temporal changes in object characteristics, includingwhen and how the mixed reality object is to be removed from the currentsession.

Any of the features described herein may be executed by an applicationfor providing a 3D environment responsive to user input that produces VRoutput for an immersive VR headset or the like. FIG. 7 is a schematicdiagram illustrating one type of an immersive VR stereoscopic displaydevice 700 may be provided in various form factors, of which device 700provides but one example. The innovative methods, apparatus and systemsare not necessarily limited to a particular form factor of immersive VRdisplay, but may be used in a video output device that enables the userto control a position or point of view of video content playing on thedevice. Likewise, a VR or AR output device may manage an audio positionor point of view of audio content playing on the device. The immersiveVR stereoscopic display device 700 represents an example of a relativelylow-cost device designed for consumer use.

The immersive VR stereoscopic display device 700 may include a tabletsupport structure made of an opaque lightweight structural material(e.g., a rigid polymer, aluminum or cardboard) configured for supportingand allowing for removable placement of a portable tablet computing orsmartphone device including a high-resolution display screen, forexample, an LCD display. This modular design may avoid the need fordedicated electronic components for video output, greatly reducing thecost. The device 700 is designed to be worn close to the user’s face,enabling a wide field of view using a small screen size such astypically found in present handheld tablet computing or smartphonedevices. The support structure 726 may provide a fixed mounting for apair of lenses 722 held in relation to the display screen 712. Thelenses may be configured to enable the user to comfortably focus on thedisplay screen 712 which may be held approximately one to three inchesfrom the user’s eyes.

The device 700 may further include a viewing shroud (not shown) coupledto the support structure 726 and configured of a soft, flexible or othersuitable opaque material for form fitting to the user’s face andblocking outside light. The shroud may be configured to ensure that theonly visible light source to the user is the display screen 712,enhancing the immersive effect of using the device 700. A screen dividermay be used to separate the screen 712 into independently drivenstereoscopic regions, each of which is visible only through acorresponding one of the lenses 722. Hence, the immersive VRstereoscopic display device 700 may be used to provide stereoscopicdisplay output, providing a more realistic perception of 3D space forthe user. Two separate displays can also be used to provide independentimages to the user’s left and right eyes respectively. It should beappreciated that the present technology may be used for, but is notnecessarily limited to, stereoscopic video output.

The immersive VR stereoscopic display device 700 may further comprise abridge (not shown) for positioning over the user’s nose, to facilitateaccurate positioning of the lenses 722 with respect to the user’s eyes.The device 700 may further comprise an elastic strap or band 724, orother headwear for fitting around the user’s head and holding the device700 to the user’s head.

The immersive VR stereoscopic display device 700 may include additionalelectronic components of a display and communications unit 702 (e.g., atablet computer or smartphone) in relation to a user’s head 730. Asupport structure 726 holds the display and communications unit 702using restraining device 724 that is elastic and/or adjustable toprovide a comfortable and secure snug fit, for example, adjustableheadgear. When wearing the support 726, the user views the display 712though the pair of lenses 722. The display 712 may be driven by theCentral Processing Unit (CPU) 703 and/or Graphics Processing Unit (GPU)710 via an internal bus 717. Components of the display andcommunications unit 702 may further include, for example, atransmit/receive component or components 718, enabling wirelesscommunication between the CPU and an external server via a wirelesscoupling. The transmit/receive component 718 may operate using anysuitable high-bandwidth wireless technology or protocol, including, forexample, cellular telephone technologies such as 3rd GenerationPartnership Project (3GPP) Long Term Evolution (LTE), Global System forMobile communications (GSM) or Universal Mobile TelecommunicationsSystem (UMTS), and/or a wireless local area network (WLAN) technologyfor example using a protocol such as Institute of Electrical andElectronics Engineers (IEEE) 802.11. The transmit/receive component orcomponents 718 may enable streaming of video data to the display andcommunications unit 702 from a local or remote video server, and uplinktransmission of sensor and other data to the local or remote videoserver for control or audience response techniques as described herein.

Components of the display and communications unit 702 may furtherinclude, for example, one or more sensors 714 coupled to the CPU 703 viathe communications bus 717. Such sensors may include, for example, anaccelerometer/inclinometer array providing orientation data forindicating an orientation of the display and communications unit 702. Asthe display and communications unit 702 is fixed to the user’s head 730,this data may also be calibrated to indicate an orientation of the head730. The one or more sensors 714 may further include, for example, aGlobal Positioning System (GPS) sensor indicating a geographic positionof the user. The one or more sensors 714 may further include, forexample, a camera or image sensor positioned to detect an orientation ofone or more of the user’s eyes, or to capture video images of the user’sphysical environment (for VR mixed reality), or both. In someembodiments, a cameras, image sensor, or other sensor configured todetect a user’s eyes or eye movements may be mounted in the supportstructure 726 and coupled to the CPU 703 via the bus 716 and a serialbus port (not shown), for example, a Universal Serial Bus (USB) or othersuitable communications port. The one or more sensors 714 may furtherinclude, for example, an interferometer positioned in the supportstructure 704 and configured to indicate a surface contour to the user’seyes. The one or more sensors 714 may further include, for example, amicrophone, array or microphones, or other audio input transducer fordetecting spoken user commands or verbal and non-verbal audiblereactions to display output. The one or more sensors may include, forexample, electrodes or microphone to sense heart rate, a temperaturesensor configured for sensing skin or body temperature of the user, animage sensor coupled to an analysis module to detect facial expressionor pupil dilation, a microphone to detect verbal and nonverbalutterances, or other biometric sensors for collecting biofeedback data.

Sensor data from the one or more sensors may be processed locally by theCPU to control display output, and/or transmitted to a server forprocessing by the server in real time, or for non-real time processing.As used herein, “real time” refers to processing responsive to userinput that controls display output without any arbitrary delay; that is,that reacts as soon as technically feasible. “Non-real time” refers tobatch processing or other use of sensor data that is not used to provideimmediate control input for controlling the display, but that maycontrol the display after some arbitrary amount of delay.

Components of the display and communications unit 702 may furtherinclude, for example, an audio output transducer 720, for example aspeaker or piezoelectric transducer in the display and communicationsunit 702 or audio output port for headphones or other audio outputtransducer mounted in headgear 724 or the like. The audio output devicemay provide surround sound, multichannel audio, so-called ‘objectoriented audio’, or other audio track output accompanying a stereoscopicimmersive VR video display content. Components of the display andcommunications unit 702 may further include, for example, a memorydevice 708 coupled to the CPU 703 via a memory bus. The memory 708 maystore, for example, program instructions that when executed by theprocessor cause the apparatus 700 to perform operations as describedherein. The memory 708 may also store data, for example, audio-videodata in a library or buffered during streaming operations. The methodsand systems described herein may be used with any suitable AR or VRequipment, including but not limited to the type of equipment describedin connection with FIG. 7 .

Further details regarding generation and use of VR environments may beas described in U.S. Provisional Pat. Applications Serial Nos.62/088,496, filed Dec. 5, 2014, and 62/330,708 filed May 2, 2016, whichare incorporated herein in their entireties by reference. For example,Provisional Pat. Application Serial No. 62/330,708 filed May 2, 2016(the “‘708 Application”) describes adjusting an AR or VR object displaybased on geometrical relationship between the objective geometry for anygiven scene and a position/orientation of each viewer wearing AR or VRgear, relative to a display screen of known size. That applicationdescribes, among other things, adjustments that may be made to renderedobject geometry to as to achieve a desired effect of an AR or VR objectrelative to a non-VR object or environment shown on a display screen,depending on the desired effect.

Mixed reality content can include content shown on a screen. A mixedreality screen is partly virtual, but is referenced to a real object inthe environment. The reference object may be, for example, the side of abuilding, a wall of a room, a display surface of a television, tablet ormobile phone, or any other object having a screen-shaped surface. Themixed reality engine can cause 2D or stereoscopic 3D content to be“projected” on such a surface, such that a grammar is needed to handlehow off-screen virtual objects relate to the onscreen objects.

To the extent that screens are used in a mixed reality session, thetechniques and algorithms for real-time computational geometry matchingfor passing objects through a screen as taught by the ′708 Applicationcan be directly applied to production of mixed reality content. Thesecomputations may be performed in real-time by the mixed reality outputdevice. Once the position of the output device to the screen, and thescreen size are defined, the geometric relationships, algorithms andmethods described can be used to define transformations (e.g., matrices)for scaling, rotating, and translating off-screen objects to positionsneeded to implement an objective or subjective transition grammars asdescribed in the ′708 Application.

Algorithms and methods described in the ′708 Application may be adaptedto similarly control object geometry with respect to any selectednon-screen reference object appearing in the user’s physicalenvironment. For example, the various different “grammars” described inthe ′708 Application may be generalized from mixed-media content withdisplay screens to mixed reality content, with or without incorporatingany display screen.

In the context of mixed reality without onscreen content, “grammar”refers to a set of rules governing interaction between a viewerexperiencing a POV into a mixed reality scene, and the physical scenethat is being mixed. These rules might vary from scene to scene (ornot), and such variation may itself be a dramatic element. Differentgrammars may be used for different entertainment objectives and mediums.For example, a fully interactive video-game like experience uses adifferent grammar than less interactive content aimed primarily atpresenting a dramatic story. One aspect of grammar for mixed-mediacontent concerns the relationship between the viewer and objects in thereal scene. For example, miniaturized virtual objects may be desirablewhen the real scene is a miniaturized physical model, depending on thedesired dramatic effect; life-sized virtual objects when the real sceneis a natural environment, and so forth. These examples illustrate thatone of the functions of mixed media grammar can be to maintain a definedgeometric relationship between a reference object in the real scene andrelated objects experienced only in VR or AR.

Mixed reality applications that do not make use of a display screen areconceptually simpler than mixed media, because the real reference objectin the scene is generally not used to convey images of scenes or objectsprojected onto it. Therefore, the relative size of the reference objectdoes not change from cut to cut of the mixed reality session. Inaddition, transition grammar - meaning control of objects transitioningbetween a screen and an AR/VR instantiation – is not generally aconcern. Instead, in most mixed-reality content, grammar will entailmaintaining a desired dramatic relationship between real referenceobjects and virtual objects, for example, life-sized, miniature, giant,growing, shrinking, and so forth. Therefore, the application process formixed reality will often entail deriving an accurate model of the user’sphysical environment relative to the user, and then scaling, positioningand moving the virtual object relative to the model as needed to achievethe desired dramatic effect relative to the current physicalenvironment, using systems and processes as described herein.

FIG. 8 shows a process or system 800 that integrates virtual contentwith real environment input to provide a mixed reality experience. Anyone or all of the illustrated elements of the process 800 may beindividually performed by each user’s immersive output equipment or(except for output of rendered immersive content) by a cinema server ornetwork. Initially, at 802, immersive digital mixed reality master datais obtained from a content selection engine as described herein aboveand decoded to obtain frame rendering parameters for each frame or setof frames. Such parameters may include, for example, selected libraryobjects appearing in the scene, position and orientation of all objectsto be rendered each associated with a set of position and orientationcoordinates that are indicated as subjective or objective, associatedobject textures for rendered objects, lighting parameters, and cameraparameters. Standard frame rendering parameters may then be adjusted foreach frame or for sets of multiple contiguous frames, as necessary.

These adjustments may include a set of viewpoint adjustments 804, 806.Each of these viewpoint adjustments may include geometric adjustments asdescribed in more detail herein below. For example, at 804, transformingobjective coordinates for indicated objects to the coordinate systemused by the applicable render engine for rendering a viewpoint.Generally, the transform 804 will transform an object’s objectivecoordinates into the coordinates used by the applicable render enginefor rendering immersive output for a particular immersive output device.The applicable render engine may be located variously, such as in arandom access memory of an immersive device, in a local auxiliary devicefor the immersive output device, in a connected server or server farm,or in a cloud computing resource. In any case, the coordinate transformwill be based on the coordinates used by the render engine andcalibration data establishing the geometrical relationship between theuser and the real scene’s objective coordinates. Any suitable transformmethod as known in the art may be used for the coordinate transform.

The adjustments may further include, at 806, transforming subjectivecoordinates for indicated objects to the coordinate system used by theapplicable render engine for rendering a viewpoint. In the trivial case,no transformation is needed because the common subjective values willwork for every render engine, and are the same for every user. However,in some cases certain transformation may be needed to put subjectivecoordinates in proper condition for rendering, for example converting toa different type of coordinate system to facilitate a particular renderengine or adding a fixed offset value to account for physicaldifferences between users participating in a multi-player session.

The adjustments may further include, at 808, adjusting a position ororientation of rendered objects based on user input, in the case ofinteractive objects. The appearance, position, or orientation ofselected objects may depend on user input. The influence of user inputmay be limited to specified objects and ranges of change, to preventdisrupting the flow of a narrative performance and maintaincontemporaneous audience members in sync.

The adjustments may further include, at 810, adjusting scene lightingparameters. In an aspect, position and orientation of scene lights maybe designated objective or subjective, and transformed as needed likeany other off screen object with respect to position and orientationcoordinated. In addition, other lighting parameters, such as intensityor color, may also be adjusted so that the brightness and color ofrendered scene elements matches the brightness and color of output onthe theater’s display screen.

The adjustments may further include, at 812, adjusting object texture,for example, applying an automatic level of detail based on a distancebetween the rendered viewpoint and each rendered object, or equivalentmeasure. Automatic level of detail provides less detailed texture mapsfor more distant objects, to improve rendering performance. Similarly,automatic level of detail adjustments may be used to select a meshdensity for off screen objects based on distance from the viewpoint,again for rendering efficiency.

The adjustments may further include, at 814, adjusting camera parametersother than position and orientation, such as focal point, field offield, and aperture, based on immersive input. Hence, an immersiverender engine may allow a user to “zoom in” or “zoom out” on the scene,with appropriate camera adjustments. Once adjustments are made therender engine may render the scene at 816 and the rendered data may bedisplayed using an immersive output device at block 818.

In view the foregoing, and by way of additional example, FIGS. 9-12 showaspects of a method 900 or methods for virtual content selection in amixed reality output process. The method 900 may be performed by an ARoutput device including a programmable computer, by a VR output deviceincluding a programmable computer, by one or more computers incommunication with the AR output device or VR output device, or by acombination of an AR or VR output device and one or more computers incommunication with the output device.

Referring to FIG. 9 , a computer-implemented method for virtual contentselection in a mixed reality output process may include, at 910, imaginga real scene encompassing a user wearing a mixed reality outputapparatus (e.g., an AR or VR headset). Further aspects of the imagingare described below in connection with FIG. 10 , and some examples areprovided above in connection with FIG. 5 . The imaging is performed inreal time, contemporaneously or immediately before a mixed realitysession executing in the mixed reality apparatus.

The method 900 may include, at 920, determining context parameter datadescribing a real context of the real scene, based on the imaging.Further examples of context determination are described in connectionwith FIG. 11 , and examples are provided in the discussion above, forexample, in connection with FIG. 5 .

The method 900 may include, at 930, selecting a set of content from acontent library, based on the data (context parameters) describing areal context, wherein the context parameters enable rendering of avirtual object in a mixed reality session by the mixed reality outputapparatus. The context parameters are provided to the mixed realitysession, which renders the virtual object based on the selectedparameters, and outputs a rendering of the virtual object in a displayof the mixed reality output apparatus. Thus, the method 900 may include,at 940, rendering the virtual object in the mixed reality session by themixed reality output apparatus, optionally based on the contextparameters.

The method 900 may include any one or more of additional operations1000, 1100, or 1200, shown in FIGS. 10-12 , in any operable order. Eachof these additional operations is not necessarily performed in everyembodiment of the method, and the presence of any one of the operations1000, 1100, or 1200 does not necessarily require that any other of theseadditional operations also be performed.

Referring to FIG. 10 showing certain additional operations 1000 forimaging a scene, the method 900 may further include, at 1010, imagingthe real scene at least in part by activating a camera coupled to themixed reality output apparatus, and receiving images of the real scenefrom the camera. The camera may be one of a pair, or one of several, inan array. The mixed reality apparatus may analyze the images from anarray based on photogrammetry, and construct a 3D model of the imagedscene based on the photogrammetric analysis of the image data from thearray. Images may be collected and analyzed periodically, for example,once per second.

In addition, the method 900 may include, at 1020, imaging the real sceneat least in part by activating a depth sensor coupled to the mixedreality output apparatus. A depth sensor may include, for example, asonar device with an emitter and receiver. Reflection of a known pulsecan be analyzed by the mixed reality apparatus, obtaining an estimateddistance to an object surface and estimated solidity of the object.

In addition, the method 900 may include, at 1030, imaging the real sceneat least in part by activating a microphone coupled to the mixed realityoutput apparatus. An encoded acoustic signal from the microphone may beanalyzed to determine acoustic conditions of the environment. Forexample, traffic noise can be used as an indicator that the user is neara busy street, and so forth.

In addition, the method 900 may include, at 1040, imaging the real sceneat least in part by activating a radio receiver coupled to the mixedreality output apparatus. The radio receiver may be tuned to receiveobject identifiers from RFID tags placed in scene objects, or to receivea broadcast from a location transmitter conveying encoded datadescribing elements of a local scene.

In addition, the method 900 may include, at 1050, imaging the real sceneat least in part by activating a triangulating locating device coupledto the mixed reality output apparatus. For example, the mixed realityapparatus may include a GPS receiver, which determines locationcoordinates of the apparatus.

Referring to FIG. 11 showing certain additional operations 1100, themethod 900 may further include, at 1110, determining the data describinga real context at least in part by characterizing a volumetric geometryof the real scene, based on the imaging. Aspects of the characterizingvolumetric geometry may be as described above in connection with FIG. 5. The volumetric geometry identifies volumes in the real environmentwith basic characteristics such as solid, liquid, air, or mixed.

In addition, the method 900 may further include, at 1120, determiningthe data describing a real context at least in part by characterizing acondition of the real scene, based on the imaging. For example, based onimage information the mixed reality apparatus may determine types,intensities and colors of virtual light used to render a virtual objectto be inserted into the scene. In addition, the apparatus may determinecurrent weather conditions, or other conditions of the real environmentthat are pertinent to characteristics of the virtual object. Forexample, if a strong wind is sensed in the real environment, the mixedreality apparatus may render the effects of the wind using awind-simulating animation loop.

In addition, the method 900 may further include, at 1130, determiningthe data describing a real context at least in part by determining anobject type, based on a characteristic volumetric geometry determinedbased on the imaging. Examples were provided in connection with FIG. 5 .A set of geometric objects may be bounded by ranges of object parametersdefining a type. The mixed reality apparatus may select the object typethat most closely matched the sensed parameter set for an object. Anyother suitable object recognition algorithm may be used.

In addition, the method 900 may further include, at 1140, determiningthe data describing a real context at least in part by determining anidentity of a scene or of an object in the scene, based on the imaging.Identity may be determined, for example, by reading identify tagsassociated with sensed objects, or by comparing location coordinates toa database of located objects (e.g., a map).

The method 900 may further include other techniques for determining thedata describing the real context. For example, the data describing thereal context may be determined at least in part by analyzing images ofthe real scene using automatic object or scene recognition algorithmsbased on 2D image input. In an alternative, or in addition, the datadescribing the real context may be determined at least in part byanalyzing images of the real scene using stereogrammetry to determine 3Dgeometry of the user’s real environment or of objects in theenvironment, and using the 3D geometry to identify the objects, type ofenvironment, or location.

Referring to FIG. 12 showing certain additional operations 1200 forselecting content to be inserted into a mixed reality session, themethod 900 may further include, at 1210, selecting the set of content(parameter set) characterizing a virtual object to be inserted into themixed reality process further based on heuristic feedback. Some examplesare provided in connection with FIG. 6 above. The heuristic feedback mayinclude, for example, implied or expressed user preference informationconcerning past objects rendered in the mixed reality session or inrelated sessions. Data recording any past mixed reality session eventsthat may be used to inform a future content choice may be collected andapplied as heuristic feedback.

In addition, the method 900 may further include, at 1220, selecting theset of content characterizing a virtual object to be inserted into themixed reality process further based on a session context of the mixedreality session. Session context means the current narrative or gamestate of the mixed reality session. Characteristics of the virtualobject may likewise be selected based on an intended effect or narrativepurpose for the game or narrative mixed reality session.

In addition, the method 900 may further include, at 1230, updating theset of content characterizing the virtual object already inserted intothe mixed reality process based on the data describing the real contextand the session context. As the real environment and session contextevolve, the virtual object’s parameters may evolve accordingly until abehavior parameter removes the virtual object from the session entirely.Until then, the virtual object may move about and react to environmentalfactors and user input in various ways.

FIG. 13 is a conceptual block diagram illustrating components of anapparatus or system 1300 for virtual content selection in a mixedreality output process, as described herein. The apparatus or system1300 may include additional or more detailed components for performingfunctions or process operations as described herein. For example, theprocessor 1310 and memory 1316 may contain an instantiation of a processfor scene and object characterization (real context determination) asdescribed herein above. As depicted, the apparatus or system 1300 mayinclude functional blocks that can represent functions implemented by aprocessor, software, or combination thereof (e.g., firmware).

As illustrated in FIG. 13 , the apparatus or system 1300 may comprise anelectrical component 1302 for imaging a real scene encompassing a userwearing a mixed reality output apparatus. The component 1302 may be, ormay include, a means for said imaging. Said means may include theprocessor 1310 coupled to the memory 1316, and to an output of a sensorarray (not shown), the processor executing an algorithm based on programinstructions stored in the memory. Such algorithm may include a sequenceof more detailed operations, for example, as described in connectionwith FIG. 10 .

The apparatus 1300 may further include an electrical component 1303 fordetermining data describing a real context of the real scene, based onthe imaging. The component 1303 may be, or may include, a means for saiddetermining. Said means may include the processor 1310 coupled to thememory 1316, the processor executing an algorithm based on programinstructions stored in the memory. Such algorithm may include a sequenceof more detailed operations, for example, as described in connectionwith FIGS. 5 or 11 .

The apparatus 1300 may further include an electrical component 1304 forselecting a set of content from a content library, based on the datadescribing a real context. The component 1304 may be, or may include, ameans for said selecting. Said means may include the processor 1310coupled to the memory 1316 and to a sensor (not shown), the processorexecuting an algorithm based on program instructions stored in thememory. Such algorithm may include a sequence of more detailedoperations, for example, as described in connection with FIGS. 6 or 12 .

The apparatus 1300 may further include an electrical component 1305 forrendering the virtual object in the mixed reality session by the mixedreality output apparatus, optionally based on the context parameters.The component 1305 may be, or may include, a means for said rendering.Said means may include the processor 1310 coupled to the memory 1316,the processor executing an algorithm based on program instructionsstored in the memory. Such algorithm may include a sequence of moredetailed operations, for example, selecting a rendering engine,providing object parameters to the rendering engine, executing therendering engine to produce an image, and sending the image for displayon an output component of the mixed reality apparatus.

The apparatus 1300 may optionally include a processor module 1310 havingat least one processor. The processor 1310 may be in operativecommunication with the modules 1302-1305 via a bus 1313 or similarcommunication coupling. In the alternative, one or more of the modulesmay be instantiated as functional modules in a memory of the processor.The processor 1310 may effect initiation and scheduling of the processesor functions performed by electrical components 1302-1305.

In related aspects, the apparatus 1300 may include a network interfacemodule (not shown) operable for communicating with system componentsover a computer network, instead of or in addition to the transceiver1312. A network interface module may be, or may include, for example, anEthernet port or serial port (e.g., a Universal Serial Bus (USB) port).In further related aspects, the apparatus 1300 may optionally include amodule for storing information, such as, for example, a memory device1316. The computer readable medium or the memory module 1316 may beoperatively coupled to the other components of the apparatus 1300 viathe bus 1313 or the like. The memory module 1316 may be adapted to storecomputer readable instructions and data for effecting the processes andbehavior of the modules 1302-1305, and subcomponents thereof, or theprocessor 1310, or the method 1200 and one or more of the additionaloperations 1000-1200 disclosed herein. The memory module 1316 may retaininstructions for executing functions associated with the modules1302-1305. While shown as being external to the memory 1316, it is to beunderstood that the modules 1302-1305 can exist within the memory 1316or an on-chip memory of the processor 1310.

The apparatus 1300 may include a transceiver 1312 configured as awireless transmitter/receiver, or a wired transmitter/receiver, fortransmitting and receiving a communication signal to/from another systemcomponent such as, for example, an RFID tag or location informationtransmitter. In alternative embodiments, the processor 1310 may includenetworked microprocessors from devices operating over a computernetwork. In addition, the apparatus 1300 may include a stereoscopicdisplay or other immersive display device 1314 for displaying immersivecontent. The stereoscopic display device 1314 may be, or may include,any suitable stereoscopic AR or VR output device as described hereinabove, or as otherwise known in the art.

Those of skill would further appreciate that the various illustrativelogical blocks, modules, circuits, and algorithm steps described inconnection with the aspects disclosed herein may be implemented aselectronic hardware, computer software, or combinations of both. Toclearly illustrate this interchangeability of hardware and software,various illustrative components, blocks, modules, circuits, and stepshave been described above generally in terms of their functionality.Whether such functionality is implemented as hardware or softwaredepends upon the particular application and design constraints imposedon the overall system. Skilled artisans may implement the describedfunctionality in varying ways for each particular application, but suchimplementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

As used in this application, the terms “component”, “module”, “system”,and the like are intended to refer to a computer-related entity, eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component or a module may be, but are notlimited to being, a process running on a processor, a processor, anobject, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component or a module. One or morecomponents or modules may reside within a process and/or thread ofexecution and a component or module may be localized on one computerand/or distributed between two or more computers.

Various aspects will be presented in terms of systems that may include anumber of components, modules, and the like. It is to be understood andappreciated that the various systems may include additional components,modules, etc. and/or may not include all of the components, modules,etc. discussed in connection with the figures. A combination of theseapproaches may also be used. The various aspects disclosed herein can beperformed on electrical devices including devices that utilize touchscreen display technologies, heads-up user interfaces, wearableinterfaces, and/or mouse-and-keyboard type interfaces. Examples of suchdevices include VR output devices (e.g., VR headsets), AR output devices(e.g., AR headsets), computers (desktop and mobile), smart phones,personal digital assistants (PDAs), and other electronic devices bothwired and wireless.

In addition, the various illustrative logical blocks, modules, andcircuits described in connection with the aspects disclosed herein maybe implemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general purpose processor may be a microprocessor,but in the alternative, the processor may be any conventional processor,controller, microcontroller, or state machine. A processor may also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration.

Operational aspects disclosed herein may be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, digital versatile disk (DVD),Blu-ray™, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such the processorcan read information from, and write information to, the storage medium.In the alternative, the storage medium may be integral to the processor.The processor and the storage medium may reside in an ASIC. The ASIC mayreside in a client device or server. In the alternative, the processorand the storage medium may reside as discrete components in a clientdevice or server.

Furthermore, the one or more versions may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedaspects. Non-transitory computer readable media can include but are notlimited to magnetic storage devices (e.g., hard disk, floppy disk,magnetic strips, or other format), optical disks (e.g., compact disk(CD), DVD, Blu-ray™ or other format), smart cards, and flash memorydevices (e.g., card, stick, or other format). Of course, those skilledin the art will recognize many modifications may be made to thisconfiguration without departing from the scope of the disclosed aspects.

The previous description of the disclosed aspects is provided to enableany person skilled in the art to make or use the present disclosure.Various modifications to these aspects will be readily apparent to thoseskilled in the art, and the generic principles defined herein may beapplied to other embodiments without departing from the spirit or scopeof the disclosure. Thus, the present disclosure is not intended to belimited to the embodiments shown herein but is to be accorded the widestscope consistent with the principles and novel features disclosedherein.

In view of the exemplary systems described supra, methodologies that maybe implemented in accordance with the disclosed subject matter have beendescribed with reference to several flow diagrams. While for purposes ofsimplicity of explanation, the methodologies are shown and described asa series of blocks, it is to be understood and appreciated that theclaimed subject matter is not limited by the order of the blocks, assome blocks may occur in different orders and/or concurrently with otherblocks from what is depicted and described herein. Moreover, not allillustrated blocks may be required to implement the methodologiesdescribed herein. Additionally, it should be further appreciated thatthe methodologies disclosed herein are capable of being stored on anarticle of manufacture to facilitate transporting and transferring suchmethodologies to computers.

1-19. (canceled)
 20. A method by a mixed reality apparatus for operatinga mixed reality session playing an immersive interactive game, themethod comprising: determining context parameter data for a real scene,based on imaging the scene by a mixed reality output apparatus;determining a session context indicating a current game state of aninteractive game playing in a mixed reality session, wherein the currentgame state comprises an indication of user progress in the immersiveinteractive game; selecting content characterizing a virtual object tobe inserted into the mixed reality session, by a selection algorithmthat eliminates prospective content from selection based on the contextparameter data and the session context, wherein the selecting comprisesselecting a class of content based on the context parameter data and thesession context; and rendering a virtual object in the mixed realitysession by the mixed reality output apparatus, based on the class ofcontent and on the data describing a real context.
 21. The method ofclaim 20, further comprising activating a device coupled to the mixedreality output apparatus, wherein the device is selected from a groupconsisting of a camera, a depth sensor, a microphone, a radio receiver,or a triangulating locating device.
 22. The method of claim 20, furthercomprising randomly selecting a particular instance of the content fromthe class of content.
 23. The method of claim 20, further comprisingdetermining the class of content, based on a characteristic volumetricgeometry determined based on the imaging.
 24. The method of claim 20,further comprising determining an identity of a scene or of an object inthe scene, based on the imaging.
 25. The method of claim 20, whereinselecting the content characterizing a virtual object to be insertedinto the mixed reality session is further based on heuristic feedback.26. The method of claim 20, wherein selecting the content characterizinga virtual object to be inserted into the mixed reality session based ona session context of the mixed reality session further comprisesmanaging exit of the virtual object from the virtual reality sessionbased on the session context and real context.
 27. The method of claim20, wherein further comprising updating the content characterizing thevirtual object already inserted into the mixed reality session based onthe data describing the real context and the session context.
 28. Anapparatus for selecting content for a mixed reality session playingimmersive interactive game content, comprising: a processor, a memorycoupled to the processor, and a stereoscopic display device coupled tothe processor, wherein the memory holds instructions that when executedby the processor, cause the apparatus to perform: determining contextparameter data for a real scene, based on imaging the scene by a mixedreality output apparatus; determining a session context indicating acurrent game state of an interactive game playing in a mixed realitysession, wherein the current game state comprises an indication of userprogress in the immersive interactive game; selecting contentcharacterizing a virtual object to be inserted into the mixed realitysession, by a selection algorithm that eliminates prospective contentfrom selection based on the context parameter data and the sessioncontext, wherein the selecting comprises selecting a class of contentbased on the context parameter data and the session context; andrendering a virtual object in the mixed reality session by the mixedreality output apparatus, based on the class of content and on the datadescribing a real context.
 29. The apparatus of claim 28, wherein thememory holds further instructions for activating a device coupled to themixed reality output apparatus, wherein the device is selected from agroup consisting of a camera, a depth sensor, a microphone, a radioreceiver, or a triangulating locating device.
 30. The apparatus of claim28, wherein the memory holds further instructions for randomly selectinga particular instance of the content from the class of content.
 31. Theapparatus of claim 28, wherein the memory holds further instructions fordetermining the class of content, based on a characteristic volumetricgeometry determined based on the imaging.
 32. The apparatus of claim 28,wherein the memory holds further instructions for determining anidentity of a scene or of an object in the scene, based on the imaging.33. The apparatus of claim 28, wherein the memory holds furtherinstructions for selecting the set of content characterizing a virtualobject to be inserted into the mixed reality session further based onheuristic feedback.
 34. The apparatus of claim 28, wherein the memoryholds further instructions for selecting the set of contentcharacterizing a virtual object to be inserted into the mixed realitysession is based on a session context of the mixed reality sessionmanaging exit of the virtual object from the virtual reality sessionbased on the session context and real context.
 35. The apparatus ofclaim 28, wherein the memory holds further instructions for updating theset of content characterizing the virtual object already inserted intothe mixed reality session based on the data describing the real contextand the session context.
 36. A non-transitory computer-readable medium,encoded with instructions that, when executed by a processor, cause anapparatus for selecting content for a mixed reality session playingimmersive interactive game content to perform: determining contextparameter data for a real scene, based on imaging the scene by a mixedreality output apparatus; determining a session context indicating acurrent game state of an interactive game playing in a mixed realitysession, wherein the current game state comprises an indication of userprogress in the immersive interactive game; selecting contentcharacterizing a virtual object to be inserted into the mixed realitysession, by a selection algorithm that eliminates prospective contentfrom selection based on the context parameter data and the sessioncontext, wherein the selecting comprises selecting a class of contentbased on the context parameter data and the session context; andrendering a virtual object in the mixed reality session by the mixedreality output apparatus, based on the class of content and on the datadescribing a real context.